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FOREWORD 


The  Intelligence  S /stems  '.Vork  Unit  within  the  U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  (ARI)  is  concerned  with  the  functions  of  human  information 
analysis,  processing,  aggregation,  integration  and  consequent  product  utilization  in  intelligence 
systems.  One  of  the  major  objectives  is  to  provide  research  findings  by  which  performance  of 
these  functions  can  be  enhanced.  One  resulting  requirement  is  to  determine  how  human 
capabilities  can  be  utilized  to  enable  the  intelligence  information  processing  system  to 
function  with  increased  effectiveness.  The  entire  research  effort  is  responsive  to  requirements 
of  RDTE  Project  201621QA754,  "Intelligence  Information  Porcessing,"  FY  1973  Work 
Program  and  to  special  requirements  of  the  U,  v  Army  Intelligence  Center  and  School. 

The  U.  S.  Army  currently  has  under  development  intelligence  information  processing 
systems  designed  to  maximize  combat  effectiveness  by  optimal  utilization  of  human 
capabilities  augmented  by  computer  support.  The  present  publication  describes  one  effort 
which  provides  data  for  more  effectively  evaluating  mar/s  capabilities  and  limitations  in 
intelligence  processing. 


J.  EXIHLANER 
Technical  Director 
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BRIEF 


Requirement* 

Judgments  of  the  probable  accuracy  of  intelligence  data  and  products  are  integral  to  the 
intelligence  process  Accurate  communication  of  the  probability  oi  uncertainty  implicit  in 
these  juogments  is  requisite  to  the  effective  production  and  utilization  of  intelligence.  Factors 
which  may  impact  on  the  accurate  communication  of  these  values  must  be  identified  and 
their  degree  of  impac1  ascertained  to  provide  a  rational  basis  for  th^  development  of 
improved  procedures  Determination  of  whether  the  encoding  of  qualitative  expressions  is 
influenced  by  context  or  group  membership  and  the  form  of  the  probability  scale  used  to 
encode  uncertainty  is  directly  relevant  to  that  goal 


Procedure 

Twenty-e.ght  subjects,  14  U  S.  Army  ennsted  men  and  14  extension  college  students, 
numerically  encoded  on  a  0  to  100  scale  each  of  15  probability  phrases  in  each  of  three 
sentence  contexts.  The  15  phrases  systematically  covered  a  wide  range  of  probabilistic 
meaning  and  the  three  sentence  contexts  involved  a  weather  forecast,  a  prediction  of 
persona!  success,  and  an  intelligence  report. 


Findings. 

1.  There  were  no  significant  differences  in  encoding  of  probability  phrases  into 
numerical  equivalents  among  the  three  sentence  contexts,  between  enlisted  personnel  and 
college  students,  or  as  a  function  of  age,  se>,  or  educational  level, 

2.  Individual  were  relatively  consistent  in  their  encoding  of  ge'en  phrases,  but  differed, 
often  radical'y,  fiom  other  individuals. 

3.  Individuals'  numerical  encodings  indicated  the  use  of  an  underlying  asymmetric 
probabihty  scak  comprised  of  a  small  number  of  intervals. 


Application  of  Findings: 

The  findings  indicate  the  use  of  qualitative  expressions  to  communicate  the  accuracy  or 
relative  likelihood  of  occurrence  of  intelligence  data  and  products  will  often  result  in  a  high 
degree  of  misunderstanding,  Personnel  involved  in  rne  production  and  use  of  intelligence 
should  be  extremely  warv  of  attempting  to  infer  numerical  values  from  qualitative 
expressions  of  uncertainty.  Difteier>:e$  in  the  encoding  of  qualitative  expressions  do  not 
appear  dependent  upon  any  of  the  general  factors  evaluated  m  this  study  other  than  the 
ambiguities  of  qualitative  phrases  themselves  mfluenced  by  individual  differences  of 
unknowr  sources. 

Further  research  should  investigate  the  use  of  standardized  lexicons  with  a  small  number 
uf  eApirbituiis  ur  ihe  duer/i  use  of  numei.Cdi  scales.  Numerical  scales  appear  to  be  the  most 
promising  and  would  facilitate  the  use  of  the  tools  tf  probability  theory  and  decision  +heory 
in  intelligence  analysis.  t 
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INTRODUCTION 

The  weather  forecast  states  Mrain  is  likely  within  the  next  24 
hours. "  An  intelligence  estimate  states  "an  enemy  attack  is  likely 
within  the  next  24  hours."  What  do  these  statements  mean?  If  the 
source  of  each  statement  were  to  express  the  forecast  in  terms  of  a 
numerical  probability,  would  the  numbers  they  provide  be  identical? 
Would  our  interpretation  of  these  statements  agree  with  the  interpre¬ 
tation  intended  by  the  source?  The  answer  to  both  questions  is 
"likely"  to  be  no.  Yet  reliable  and  valid  communication  of  the  degree 
of  uncertainty  in  forecasted  events  can  be  of  great  practical  impor  - 
tance . 


Intelligence  is  seldom  perfect  and  evaluations  containing  inade¬ 
quate  data  and  doubtful  conclusions  can  often  be  extremely  useful.  The 
user  of  an  intelligence  evaluation  will  naturally  be  influenced  by  the 
degree  of  certainty  which  has  been  attached  to  it  A-*  It  is  the  respon¬ 
sibility  of  the  intelligence  officer  to  determine  the  degree  of  un¬ 
certainty  of  a  given  statement  and  then  to  communicate  this  to  the  users 
of  the  intelligence..  It  should  be  noted  that  the  problem  of  communicat¬ 
ing  uncertainty  along  such  dimensions  as  source  reliability  and  infor¬ 
mation  accuracy  is  a  problem  within  the  intelligence  section  ao  well 
as  for  users. 

However,  there  is  no  standard  terminology  in  use  to  describe  the 
probability  of  occurrence  of  forecasted  events.  Sherman  Kent  in  the 
mid  1550’s  proposed  a  list  of  words  and  phrases  to  be  associated  with 
specified  ranges  of  probability  to  ensure  their  understanding  (Table  1) . 
The  motivation  behind  Kent’s  proposal  was  his  observation  of  large 
differences  in  the  probability  values  assigned  to  given  phrases  by  dif¬ 
ferent  intelligence  analysts.  Kent's  chart  was  never  formally  adopted 
by  any  intelligence  agency  2^. 

More  formal  studies  of  the  assignment  of  numerical  probability 
values  to  probability  phrases  or  words  have  also  found  large  individual 


A-/  Samet,  M.  G.  Checker  confidence  statements  as  affected  by  performance 
of  initial  image  interpreter.  ARI  Technical  Research  Note  214. 

September  1969. 

2-/  Samet,  M.  G.  Subjective  interpretation  of  the  source  reliability  and 

information  accuracy  rating  scales.  ARI  Technical  Paper,  1073  ( in  press] , 

2^  Platt,  W.  Strategic  intelligence  production.  New  York:  Praeger,  1957. 


-I  * 


_  V 


X! 

GJ 

u 

D 

£ 

o 

X 

X 

■3 

u 

0 

a 

CO 

c 

OJ 

c 

CO 

o 

£ 

o 

cO 

xi 

>> 

D 

u 

X> 

X 

>> 

CO 

0 

a; 

0 

QJ 

c 

X 

CO 

o 

X 

<u 

X 

X 

X 

•H 

QJ 

CO 

X 

X 

u 

X 

cO 

X 

X 

OJ 

X 

CO 

X 

•H 

qj 

o 

<d 

0 

X 

a 

cO 

c 

M 

X 

<0 

X 

X 

X 

a 

O 

X 

U 

o 

QJ 

C 

X 

o 

X 

C 

QJ 

X 

CO 

O 

OJ 

X 

tO 

to 

00 

cO 

X! 

a 

(0 

c 

M 

X) 

X 

0 

<1) 

X 

X 

x 

o 

o 

cO 

X 

o 

X 

U 

CO 

X 

a 

OJ 

X 

to 

0) 

3 

c 

o 

X 

0 

00 

cO 

a 

o 

X 

X 

£ 

60 

X 

(0 

o 

a; 

a  x 

CO 

0 

O 

X 

*H 

CO 

X 

X 

& 

< 

CO 

X 

2; 

5S 

< 

X 

CO 

<U  CO 


X 

3 

o 

X) 

<U  X 
>  a  OJ 
QJ  (U  H 
•H  XJ  X 
H  *H  i! 
Q)  >  *H 
«  W  X 


X  >>  >>  M 
cO  X  rH  cO 

X  M  <D  GJ 

O  X  ^  a 

m  03  r?  a 

pH  tn  x  < 


X  O  X  ^ 

X  M  >%  0)  X 

cO  an  X  X 

X  MX 

o  x  x  x  x 

U  O  03  O 

Pn  S3  Pm  X  SS 


X  X  J3 

a)  >  <x 
x  aj  x 
x 

x  x  3 
o  o  o 
JZ  Z  Q 


X 

X 

x  a) 

X  60 
X  U 

to  to 

X  X 

o 

M 

Ph 


ON 

-<r 

CTv 

cr> 

o 

CJ> 

00 

m 

ro 

o 

1 

! 

1 

1 

X 

m 

O 

o 

n 

oo 

X 

X 

Modified  from  Platt  ( 1  357)  pp  208-21 0 


differences  among  subjects  i  r  _  1  :  .  An  informal  study  con¬ 

ducted  by  NATO-*  provided  similar  results  for  intelligence  statements. 
Twenty-three  officers  of  10  different  nationalities,  all  fluent  in 
English  and  with  a  background  in  intelligence,  described  in  terms  of 
"chances  out  of  100"  what  a  series  of  statements  meant  to  them.  The 
size  of  the  range  of  numerical  values  assigned  to  specific  phrases 
varied  from  25  to  80.  Another  ir'orma!  study  found  differences  as 
large  as  50  between  the  numerical  values  assigned  to  an  intelligence 
evaluation  by  its  two  authors1—' 

An  understanding  of  the  basis  for  the  large  individual  differences 
in  the  numerical  encoding  of  qualitative  expressions  of  uncertainty 
should  aid  in  designing  better  methods  for  communicating  degrees  of 
uncertainty.  These  methods  may  take  the  form  of  a  glossary  as  suggested 
by  Kent,  a  change  to  direct  numerical  estimation  13  i3  or  some  combi¬ 
nation  of  these. 


Cohen,  J.  ,  Dearnley,  E.  J.,  &  Hansel,  C.  E.  M.  A  quantitative  study 
of  meaning.  British  Journal  of  Educational  Psychology,  1958,  28_,  141-148. 

rl  Levine,  J.M.,  &  Eldridge,  D.  The  effects  of  ancillary  informal  ion  upon 
photointerpreter  performance.  Washington,  D.  C.:  American  Institutes 
for  Research,  Report  Number  AIR-20131-12/70FR,  December  1970. 

Lichtenstein,  S.,  &  Newman,  J.  R.  Empirical  scaling  of  common  verbal 
phrases  associated  with  numerical  probabilities.  Psychonomic  Science, 
1967,  9,  563-564. 

Z.  Samet,  Subjective  interpretation,  1973. 

Simpson,  R.  H.  Stability  in  meanings  for  quantitative  terms:  A  com¬ 
parison  over  20  years.  Quarterly  Journal  of  Speech,  1963,  49^,  146-151. 

2Ls  Stone,  D.  R.,  &  Johnson,  R.  T.  A  study  of  words  indicating  frequency. 
Journal  of  Educational  Psychology,  1959,  5J3,  224-227. 

13  Letter,  MAS  (Army)  (69)  559,  from  NATO  Assistant  Chief  of  Staff  for  In¬ 
telligence  to  Military  Agency  for  Standardization,  OTAN/NATO,  Autoronte 
Brussels/Zaventem  B-1110,  Brussels  39,  Belgium;  dated  20  February  19/0, 
Subject:  Proposed  Agenda  Item  for  Next  Meeting  of  the  Intelligence  Pro¬ 
cedures  Inter-service  Working  Party  (NU) . 

13  Kelly,  C.  W. ,  III,  &  Peterson,  C.  R.  Probability  estimates  and  proba¬ 
bilistic  procedures  in  current-intelligence  analysis.  Report  FSC  71- 
5047  Federal  Systems  Division,  IBM  Corporation,  Gaithersburg,  Maryland, 
June  1971. 

12 

Ibid. 


Samet,  Subjective  interpretation,  1973. 
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The  present  study  was  designed  to  explore  the  potential  sources  of 
individual  differences  in  numerical  encoding:  sentence  context  and 
group  membership  of  individuals.  Previous  work1-4'  suggests  that  context 
influences  the  encoding  of  probability  phrases,  but  did  not  assess  the 
significance  of  context  effect:.  Group  membership  may  also  play  a  role 
in  numerical  encoding  due  to  the  experience  and  training  which  dif¬ 
ferentiates  groups  of  individuais.  An  additional  interest  in  the  present 
study  is  the  nature  of  the  scale  used  by  individuals  in  mapping  prob¬ 
ability  phrases  into  numerical  equivalents. 


Objectives 

1.  To  assess  the  effect  of  context  on  the  numerical  eroding  of 
qualitative  statements  of  probability. 

2.  T ^  compare  numerical  assignments  to  probability  phrases  made 
by  military  personnel  and  by  evening  lege  students. 

3.  To  determine  the  consistency  and  the  form  of  the  probability 
scale  used  in  assigning  numerical  estimates  to  probability  phrases. 


,  METHOD 


Subjects 

Two  groups  of  subjects  were  used.  The  first  group  consisted  of  14 
U.S.  Army  enlisted  men  who  had  recently  completed  training  as  image  in¬ 
terpreters,  All  had  scored  above  the  mean  for  all  enlisted  men  on  the 
Army’s  general  technical  aptitude  test.  The  second  group  consisted  of 
14  students  in  an  introductory  psychology  course  at  the  Graduate  School 
of  the  U.S,  Department  of  Agriculture.  All  had  a  high  school  diploma, 
and  the  mean  educational  level  was  14.2  years  of  school  with  a  range  of 
12  to  20  years. 


Experimental  Materials 

A  questionnaire  with  43  sentences,  the  factorial  combination  of  15 
probability  words  or  phrases  and  three  sentence  contexts,  was  prepared. 

The  15  phrases  were  chosen  from  among  those  used  in  prior  studies  i^ig/ 
to  systenatically  cover  a  wide  range  of  probabilistic  meaning,  (Appendix). 


-/  Cohen,  De/rnley,  a  Hansel,  1958. 
^  Levine  &  Eldridge,  1970. 

^  Lichtenstein  &  Newman,  1967. 
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Eight  of  these  phrases  consisted  of  a  systematic  variation  of  adverbs 
combined  with  the  root  "likely"  and  four  were  combinations  of  adverbs 
with  the  root  "probable."  The  three  sentences  were  selected  to  provide 
different  contexts  as  illustrated  below:' 

The  official  weather  forecast  says  that  rain  is  highly 
probable  for  tomorrow. 

You  tell  someone  that  it  is  likely  you  will  win  a 

contest . 

The  CIA  reports  that  from  satellite  photographs  it  is 
very  probable  that  anti-missile  sites  are  being  construct¬ 
ed  around  Moscow, 

The  sentences  were  presented  in  stimulus-response  pairs  composed  of  a 
stimulus  sentence  using  one  of  the  15  probability  phrases  and  a  response 
sentence  in  which  the  subject  encoded  the  probability  phrase  into  the 
number  of  chances  out  of  100  which  most  clearly  reflected  the  degree  of 
uncertainty  implied  by  the  sentence. 

The  questionnaire  was  arranged  in  a  booklet  w’ith  each  page  contain¬ 
ing  three  sentences.  The  three  sentence  contexts  appeared  in  a  random 
orde'.  on  each  page,  and  a  given  probability  phrase  appeared  only  once 
on  any  page.  Each  subject  received  a  different  booklet,  i.e.,  a  dif¬ 
ferent  random  order  of  the  45  sentences. 


Procedure 

Subjects  were  given  the  questionnaire  in  three  groups:  the  14 
college  students,  and  two  groups  of  enlisted  men  of  8  and  6  subjects, 
respectively.  Each  subject  was  given  a  copy  of  the  printed  instructions 

This  is  a  study  to  determine  the  meaning  of  some  common 
words  for  certainty.  In  the  booklets  you've  received, 
you  will  find  pairs  of  sentences  like  the  following  set: 

The  official  weather  forecast  states  that  rain  is  somewhat 
likely  tomorrow. 

This  means  there  are  _  chances  out  of  100  of  rain  tomor¬ 

row. 

In  the  second  sentence  you  should  place  a  number  from  0  to 
100  describing  the  degree  of  certainty  you  think  the  sen¬ 
tence  indicates.  For  example,  in  the  sentence  above  I  would 
put  "79,"  indicating  the  sentence  means  to  me  that  there  are 
79  chances  out  of  100  of  rain  tomorrow.  The  weather  fore¬ 
caster  may  have  intended  to  indicate  there  are  70  chances 
out  of  100  of  rain  tomorrow.  Your  answer  may  not  agree  with 
either  mine  or  the  forecaster's. 
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There  are  n<o  right  answers.  You  should  consider  each  sen¬ 
tence  separately  and  choose  a  number  which  best  describes 
the  certainty  you  think  the  sentence  indicates.  If  you  are 
not  sure  what  number  to  use,  use  the  first  number  between  0 
and  ICO  that  comes  to  your^ind.. 

After  the  experimenter  read  the  instructions  aloud  and  answered  c.ny 
questions,  each  subject  completed  the  questionnaire.;  The  entire  session 
required  about  30  minutes  for  each  group. 


RESULTS 

A  three-way  analysis  of  variance  of  subjects’  responses,  subject 
group  x  probability  phrase  x  sentence  context,  revealed  no  significant 
main  effects  between  military  and  college  subjects  or  among  the  three 
sentence  contexts.  A  significant  three-way  interaction  of  subject 
groups,  probability  phrases,  and  sentence  context,  F(28,728)  =  1.70, 
p .03,  reflects  the  large  differences  between  probability  phrases 
and  som°  small  differences  in  the  encoding  of  specific  probability 
phrases  in  particular  sentence  contexts  by  military  and  college  sub¬ 
jects,  There  were  significant  differences  among  the  15  probability 
phrases,  F(14,364)  =  197.06,  p<  .001.  The  differences  among  probabil¬ 
ity  phrases  account  for  over  68%  of  the  total  variance  while  the  signifi¬ 
cant  interaction  accounts  for  less  than  one  percent  of  the  total  variance 
Using  Scheffe’s  procedure  for  post  hoc  mean  comparisons  13,  the  critical 
difference  between  mean  responses  on  probability  phrases  is  12.7  at  the 
0.05  level  of  significance.  The  fifteen  probability  phrases  fall  into 
three  clusters  using  Scheffe’s  criterion:  phrases  ranked  1-3,  5-9, 

10-15  (Table  2),  The  phrase  ranked  fourth,  "quite  likely /’  falls  between 
the  first  two  clusters  end  is  not  significantly  different  from  phrases 
in  either  cluster.  With  this  latter  phrase  as  an  exception,  phrases 
within  a  cluster  are  not  significantly  different  from  each  other,  but  are 
significantly  different  from  any  phrase  in  another  cluster. 

Descriptive  statistics  summarizing  the  data  on  probability  phrases 
are  shown  in  Table  2,  There  is  close  agreement  between  mean  and  median 
numerical  assignments.  For  only  three  probability  phrases  do  the  mean 
and  median  differ  by  mote  than  5.  Two  of  these  phrases,  "fairly  unlikely' 
and  "  probable"  ate  strongly  skewed  to  the  left.  The  remaining  phrases 
except  '’fairly  likely/'  "fair  chance"  and  "possible'  also  have  skewed 
distributions — witn  the  skew  toward  the  lower  end  of  the  scale.  The 
skewed  distributions  indicate  that,  depending  upon  the  error  criterion 
used,  different  "best  estimates"  (mean,  median  or  mode)  of  an  in¬ 
dividual’s  numerical  encoding  of  specific  phrases  will  be  generated. 


13  Winer,  B,  J.  Statistical  principles  in  experimental  design,  New  York: 


McGraw-Hill,  1962. 
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However,  error  criteria  of  number  of  correct  estimates,  smallest 
absolute  error,  and  smallest  signed  error  (mode,  median  and  mean, 
respectively)  each  imply  a  three-interval  scale  with  intervals  of 
0-40,  40-70,  and  70-100. 

The  mean  numerical  assignments  To  the  six-mirr or- image  pairs  of 
probability  phrases  indicate  an  underlying  asymmetry  in  the  subject1 s 
use  of  the  probability  scale.  When  attached  to  "likely adverbs  were 
ordered  as  follows:  "very'1  >  "quite"  >  no  adverb  >  "fairly./1  How¬ 
ever,  when  attached  to  "unlikely,”  adverbs  were  ordered,  "quite"  *>  "vervr  -> 
"fairly"  >  no  adverb.  When  attached  either  to  "probable"  or  to  "im¬ 
probable,”  adverbs  were  ordered  the  same  way,  that  is,  "highly" >  no 
acwerb.  This  symmetric  set  of  phrase  pairs  did  not  lead  to  a  s>mmetric 
response  set,  apparently  due  to  scale  compression  in  the  lower  half  of 
the  probability  scale.  The  mean  range  from  "fairly  likely"  to  "very 
likely"  is  27.9  whereas  the  mean  range  from  "unlikely"  to  "quite  un¬ 
likely"  which  includes  "fairly  likely"  and  "very  unlikely"  is  8.7;  the 
mean  range  from  "probable"  to  "highly  probable"  is  21.5,  whereas  the 
mean  range  from  "improbable"  to  "highly  improbable"  is  3.7.  Differ¬ 
ences  in  the  ordering  of  ad/erbs  attached  to  "unlikely"  when  compared 
to  "likely"  ure  confounded  ith  the  compression  in  the  lover  half  of 
the  probability  scale.  A  dt  rease  in  the  range  of  numbers  into  which 
a  set  of  phrases  is  mapped  c  encoded  would  increase  the  chances  for 
a  reversal  in  the  ordering  of  >hrases. 

A  hierarchical  cluster  ana. /sis  was  used  to  further  identify  the 
interrela  Lonships  between  phrasus1-^  .  The  product  moment  correlation 
was  used  as  a  measure  of  association  in  an  unweighted  analysis.  The 
association  value  within  any  set  is  the  average  correlation  computed 
from  the  original  correlation  matrix.  This  analysis  has  the  advantage 
of  showing  a  continuum  of  clusters  as  it  looks  progressively  (in  steps 
for  the  most  compact  and  isolable  clusters,  then  ^or  the  next  most  com¬ 
pact  and  so  on,  ending  with  the  whole  set.  The  results  are  displayed 
as  a  hierarchical  line  network  or  dendrogram,  where  the  length  of  a  line 
segment  joining  a  pair  of  phrases  reflects  the  level  of  association  be¬ 
tween  the  phrases  (Figure  1),  There  are  three  clusters  with  a  relative¬ 
ly  high  degree  of  intra-cluster  association:  the  first  consisting  of 
the  phrases  ranked  1  to  4,  the  second  of  the  phrases  ranked  5  to  8,  and 
the  third  of  the  phrases  ranked  9  to  12  and  14,  The  remaining  two 
phrases,  "very  unlikely"  and  "highly  improbable,"  ranked  13  and  15  are 
not  highly  correlated  with  one  another;  they  "cluster"  together  only  at 
a  relatively  low  level  of  association,  and  they  join  the  third  of  the 


15  Sneath,  P.  H.  A.,  &  Sokol,  R.  R.  Numerical  taxonomy.  New  York: 
Freeman  (in  press). 
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three  principle  word  clusters  at  an  even  lower  level.  This  is  perhaps 
due  to  the  relatively  low  inter-subject  variability  cn  these  two  phrases. 
The  degree  of  fit  between  the  derived  cluster  structure  and  the  original 
data  matrix  was  assessed  using  the  cophenetic  correlation  between  the 
original  measures  of  association  and  the  predicted  measures  of  association 
based  on  :he  dendrogram.  This  value  was  . /3,  indicating  an  acceptable 
fit.  Those  results  parallel  those  obtained  using  a  Scheffe*  criterion 
and,  considering  the  relatively  low  number  of  sui  ^acts.,  the  diff^.t  aces 
are  surprisingly  minor. 

The  percent  of  agreement  in  encoding  a  phrase  across  sentences  was 
used  as  a  measure  of  within-subject  consistency  (Table  3).  Forty-eight 
percent  of  the  phrases  were  encoded  into  exactly  the  same  number  for  at 
least  two  of  the  three  sentences  and  on  eight  percent  of  the  phrases 
into  the  same  number  for  all  three  sentences.  Relaxing  ch*  criterion 
of  agreement,  to  include  encoding  into  numbers  within +  5  of  each  other, 
subject  j%  encodings  agreed  on  at  least  two  of  the  three  sentences  for 
M%  of  the  phrases  and  on  27%  of  the  phrases  for  all  three  sentences. 

Thus,  subjects  on  the  average  encoded  at  least  eleven  phrases  on  two 
of  the  three  sentences,  including  four  phrases  on  all  three  sentences, 
within  1*3  of  each  ocher.  The  high  percentage  of  agreements  was  uni¬ 
formly  distributed  across  subjects:  the  standard  deviation  of  the  two 
distributions  of  pair  agreements  over  subjects  was  2.29  and  2.00  for 
exe'et  and  +5  agreements,  respectively,  and  122  and  2.20  for  the  dis¬ 
tribution  of  exact  and  1 5  agreements  on  all  three  sentences. 

Correlation  of  subjects'  responses  by  phrase  with  age,  sex,  and 
number  of  years  of  school  showed  that  all  were  non-significant.  Al¬ 
though  inter-subject  differences  account  for  more  than  five  percent  of 
the  total  variance,  none  of  the  variables  examined  in  this  study  were 
related  to  these  differences. 

The  fifteen  probability  phrases  used  in  the  present  experiment 
were  among  those  used  in  two  earlier  studies .  Although  the  nu¬ 
merical  values  obtained  for  specific  phrases  differ  in  all  three  sets 
of  results,  the  differences  in  the  numerical  encoding  are  less  than  15 
for  any  phrase  (Appendix).  Thrae  of  the  phrases  ("likely,"  "probable" 
and  "improbable")  were  used  by  Cohen,  Dearnley  and  Hansel?!'  and  the 
size  of  the  differences  from  the  present  results  and  from  previous 
studies  is  in  the  same  range,  less  than  15  for  any  phrase.  The  pattern 
or  rank  ordering  of  phrases  from  the  1967  and  1970  studies  is  also 
similar  to  the  present  data  (Kendall  coefficient  of  concordance  = 

.992,  p  <  .002). 


— /  Levine  &  Eldridge,  19  7^ • 

29  Lichtenstein  &  Newman,  1967. 

^  (’  her,  Dearnley,  &  Hansel,  1958. 
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MEAN  PERCENT  OF  ENCODING  AGREEMENTS  ACROSS  SENTENCE  CONTEXT  S 
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DISCUSSION 


The  scale  used  by  individuals  in  this  study  for  numerically  mapping 
uncertainty  phrases  was  relatively  stable  but  differed,  sometimes  radi¬ 
cally,  between  individuals.  For  example,  the  phrase  ’’fair  chance"  was 
encoded  into  exactly  the  same  number  for  two  out  of  three  sentences  by 
over  40%  of  the  subjects,  but  the  numbers  ranged  from  1  to  100.  The 
data  indicated  that  the  mapping  was  not  influenced  by  sentence  context 
or  by  the  individual’s  group  membership.  Further,  there  was  no  finding 
of  a  significant  correlation  between  numerical  assignments  and  the  age, 
the  sex,  or  the  educational  level  of  the  individual.  Thus,  although  an 
individual's  probability  scale  may  be  stable,  neither  the  weather  fore¬ 
caster's  nor  the  intelligence  assessor's  use  of  a  probability  phrase 
would  be  likely  to  agree  with  a  user's  numerical  interpretation  of  the 
phrase  or  with  each  other's.  This  suggests  that  an  individual's  encod¬ 
ing  of  probability  phrases  might  be  used  as  an  indicator  of  other  char¬ 
acteristics  such  as  risk-taking. 

If  a  probability  scale  were  labeled  in  accord  with  a  typical  sub¬ 
ject's  numerical  encoding,  the  resulting  scale,  although  veridical  to 
the  subject's  reported  impressions,  might  look  so  peculiar  as  to  confuse 
or  mislead  both  subjects  and  experimenters  (Figure  I).  The  scale  would 
be  as>xnmetric  to  reflect  the  asymmetry  between  mirror-image  pairs  and 
compressed  for  values  beiow  0.5.  It  would  also  be  a  discrete  scale  with 
perhaps  three  intervals  in  addition  to  the  anchor  points  of  "impossible” 
and  "certain"  events.  These  three  intervals  would  be  shifted  toward 
the  upper  half  of  the  scale  to  reflect  the  lack  of  differentiation  in  the 
lower  half.  Thus,  phrases  attached  to  points  on  the  probability  scale 
to  facilitate  an  individual's  understanding  of  the  scale  may  in  fact 
be  confusing  to  him.  This  conclusion  is  also  suggested  by  the  fact 
that  individuals  often  claim  that  numerically  reported  subjective  prob¬ 
abilities  do  not  fit  their  verbal  conceptualizations. 

The  use  of  a  standardized  lexicon  with  a  small  number  of  expres¬ 
sions  or  the  direct  use  of  numerical  scales  to  communicate  degrees  of 
uncertaint}  should  be  investigated.  Numerical  encoding  provides  an  index 
of  the  success  of  communicating  degrees  of  uncertainty  using  qualitative 
expressions.  The  present  results  and  those  of  earlier  studies 


23  Levine  &  Eidridge,  1970. 

^  Lichtenstein  &  Newman,  1967. 

Si 

Rigby,  L.  V.,  and  Swain,  A.  D.  In-flight  target  reporting — How  much 
is  "a  bunch"?  Human  Factors,  1971,  13^,  177-182. 

53 

Samet,  Subjective  interpretation,  1973. 
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suggest  that  the  variability  within  and  between  individuals  in  inter¬ 
preting  qualitative  expressions  is  so  high  as  to  often  result  in  a 
significant  degree  of  misunderstanding.  Further,  the  increased  use 
cf  the  tools  of  probability  and  decision  tneory  anticipated  with  the 
development  of  ARTADS  (Army  Tactical  Data  Systems),  particularly  TOS 
(Tactical  Operations  System),  will  require  numerical  values.  This 
strongly  suggests  that  numerical  scales  rather  than  a  lexicon  of 
qualitative  phrases  are  the  most  promising  method  for  improving  com- 
munication  of  tne  degree  of  uncertainty  in  intelligence  data  and  pro¬ 
ducts. 


CONCLUSIONS 

The  conclusions  of  prime \ pie  interest  in  identifying  sources  of 
individual  differences  in  the  numerical  encoding  of  probability  phrases 
are: 


1.  Encoding  of  probability  phrases  into  numerical  equivalents  was 
not  influenced  by  sentence  context. 

2.  Encoding  of  probability  phrases  into  numerical  equivalents  was 
similar  across  military  personnel  and  college  students  and  was  not  cor¬ 
related  with  age,  sex,  or  education  beyond  high  school. 

3.  Individuals  were  relatively  consistent  in  their  encoding  of  a 
given  probability  phrase,  but  are  likely  to  differ  from  other  individuals. 

4.  Individuals’  numerical  encodings  indicated  the  use  of  an  under¬ 
lying  asymmetric  probability  scale  comprised  of  a  small  number  of  inter¬ 
vals. 
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